Papers by Yong Man Ro
TroL: Traversal of Layers for Large Language and Vision Models (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing open-source LLVMs that perform comparably to closed-source models such as GPT-4V are often considered too large, having a larger number of layers. |
| Approach: | They propose a new efficient LLVM family with 1.8B, 3.8B, and 7B LLM model sizes, Traversal of Layers, which enables the reuse of layers in a token-wise manner. |
| Outcome: | The proposed model outperforms open-source models with larger model sizes and outperformed closed-source LLVMs with substantial models. |
Where Visual Speech Meets Language: VSP-LLM Framework for Efficient and Context-Aware Visual Speech Processing (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Visual speech processing requires context modeling due to the ambiguous nature of lip movements. |
| Approach: | They propose a framework to maximize the context modeling capability by bringing the power of LLMs. |
| Outcome: | The proposed framework maximizes the power of visual speech processing by bringing it to the forefront of the field. |
What if...?: Thinking Counterfactual Keywords Helps to Mitigate Hallucination in Large Multi-modal Models (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods to reduce hallucination in large multi-modal models are lacking in addressing this problem. |
| Approach: | They propose a method that implants counterfactual thinking into Large Multi-modal Models using self-generated counterfact keywords into the models. |
| Outcome: | The proposed method improves the reliability of large multi-modal models in addressing hallucination. |
CoLLaVO: Crayon Large Language and Vision mOdel (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing Large Language Models (LLMs) and instruction tuning have been used to drive the evolution of Vision Language Model (VLM) towards a versatile general-purpose model. |
| Approach: | They propose a learning strategy of Dual QLoRA to preserve object-level image understanding without forgetting it during visual instruction tuning, thereby achieving a significant leap in numerous VL benchmarks in a zero-shot setting. |
| Outcome: | The proposed model outperforms closed-source models on vision language tasks and achieves a significant leap in numerous benchmarks. |
MMS-LLaMA: Efficient LLM-based Audio-Visual Speech Recognition with Minimal Multimodal Speech Tokens (2025.findings-acl)
Copied to clipboard
| Challenge: | Recent Large Language Model (LLM) based AVSR systems incur high computational costs due to high temporal resolution of audio-visual speech. |
| Approach: | They propose an efficient multimodal speech LLM framework that minimizes token length while preserving essential linguistic content. |
| Outcome: | The proposed approach reduces token usage by 86% while using only 3.5 tokens per second. |